Phantom queue link level load balancing system, method and device

ABSTRACT

A data processing system includes a phantom queue for each of a plurality of output ports each associated with an output link for outputting data. The phantom queues receive/monitor traffic on the respective ports and/or the associated links such that the congestion or traffic volume on the output ports/links is able to be determined by a congestion mapper coupled with the phantom queues. Based on the determined congestion level on each of the ports/links, the congestion mapper selects one or more non or less congested ports/links as destination of one or more packets. A link selection logic element then processes the packets according to the selected path or multi-path thereby reducing congestion on the system.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/862,509, filed on Jan. 4, 2018, and entitled “PHANTOM QUEUE LINKLEVEL LOAD BALANCING SYSTEM, METHOD AND DEVICE” which is a continuationof U.S. application Ser. No. 14/667,568, filed on Mar. 24, 2015, andentitled “PHANTOM QUEUE LINK LEVEL LOAD BALANCING SYSTEM, METHOD ANDDEVICE” which claims priority under 35 U.S.C. § 119(e) of the U.S.provisional patent application Ser. No. 62/043,331, filed Aug. 28, 2014,and titled “PHANTOM QUEUE LINK LEVEL LOAD BALANCING SYSTEM, METHOD ANDDEVICE,” all of which are hereby incorporated by reference.

FIELD OF INVENTION

The present invention relates to load balancing. More particularly, thepresent invention relates to using phantom queues to balance the load ona system.

BACKGROUND OF THE INVENTION

Load balancing has become increasingly important as data centers look toadopt solutions to minimize congestion and/or packet loss andapplication jitter. Ethernet switches typically have static balancealgorithms that are limited because they do not response to load in thenetwork. Thus, the current switches are unable to dynamically adjust todifferent loads and are as a result not as efficient as possible.

BRIEF SUMMARY OF THE INVENTION

A data processing system comprises a phantom queue for each of aplurality of output ports each associated with an output link foroutputting data. The phantom queues receive/monitor traffic on therespective ports and/or the associated links such that the congestion ortraffic volume on the output ports/links is able to be determined by acongestion mapper coupled with the phantom queues. Based on thedetermined congestion level on each of the ports/links, the congestionmapper selects one or more non or less congested ports/links asdestination of one or more packets. A link selection logic element thenprocesses the packets according to the selected path or multi-paththereby reducing congestion on the system. As a result, the systemprovides the advantage of providing dynamic load balancing for non-TCPtraffic by leveraging the phantom queue fill levels. A first aspect isdirected to a dynamic load balancing system on a processing microchip.

The system comprises a multipath interface group comprising a pluralityof paths for outputting packets from the microchip, wherein each of thepaths is coupled to an output port of the microchip, link selectionlogic that receives input traffic packets and, for each of the packets,selects which one of the output ports the packet is to be output fromonto the path coupled to the one of the output ports and a plurality ofshapers, wherein each of the shapers is coupled to one of the outputports and limits the outputting of the packets out of the output portsuch that a rate of data output by the output port is below a dataoutput rate threshold, and further wherein each of the shapers indicatea congestion level of the output port coupled to the shaper thatcorresponds to a quantity of the packets sent to the output port by thelink selection logic during a time period, wherein for each packet thelink selection logic determines whether the packet has a transmissioncontrol protocol (TCP) format, and if the packet does not have the TCPformat, the link selection logic selects the one of the output portsbased on the congestion level of each of the output ports. In someembodiments, if the packet does have the TCP format, the link selectionlogic selects the one of the output ports independent of the congestionlevel of each of the output ports. In some embodiments, if the packetdoes have the TCP format, the link selection logic selects the one ofthe output ports based on a hash of the packet and an equal or weightedcost multipath selection protocol. In some embodiments, if the packetdoes not have the TCP format, the link selection logic selects the oneof the output ports according to a metric except the link selectionlogic will remove all of the output ports whose congestion level isabove a congestion threshold value from a pool of the output ports thatare able to be selected according to the metric. In some embodiments, ifthe packet does not have the TCP format and all of the output ports havea congestion level that is above the congestion threshold value, thelink selection logic selects the one of the output ports according tothe metric while including all of the output ports in the pool despitethe congestion level of all of the output ports. In some embodiments,the metric is one of the group consisting of round robin, random, andsmallest congestion level first. In some embodiments, each of theshapers comprise a phantom queue and a credit generator that deposits acredit into the phantom queue at a predefined credit deposit rate,wherein as each packet is output by one of the output ports, the shapercoupled to the one of the output ports removes one or more credits fromthe phantom queue of the shaper such that a total value of the removedcredits is equal to or greater than a size of the packet. In someembodiments, the link selection logic determines the congestion level ofeach of the output ports based on the number of credits within thephantom queue coupled to the output port. In some embodiments, thesystem further comprises a plurality of packet queues each coupled withone of the output ports such that the queues receive and queue each ofthe packets to be output by the output ports. In some embodiments, thelink selection logic determines the congestion level of each of theoutput ports based on a number of the packets within the packet queueassociated with the output port. In some embodiments, the system furthercomprises one of more additional shapers, wherein each of the additionalshapers is coupled to one of the output ports and monitors theoutputting of the packets out of the output port to determine whetherthe rate of data output by the output port is above an additional dataoutput rate threshold, and further wherein each of the additionalshapers indicate an additional congestion level of the output portcoupled to the additional shaper that corresponds to the quantity of thepackets sent to the output port by the link selection logic during thetime period.

A second aspect is directed to a link selection logic element stored ona non-transitory computer-readable medium of a processing microchiphaving a plurality of shapers and a multipath interface group includinga plurality of paths for outputting packets from the microchip, whereineach of the paths is coupled to an output port of the microchip and eachof the shapers is coupled to one of the output ports and monitors theoutputting of the packets out of the output port to determine whether arate of data output by the output port is above a data output ratethreshold, the link selection logic element configured to receive aplurality of traffic packets input by the microchip, for each of thetraffic packets, determine whether the packet has a transmission controlprotocol (TCP) format and for each of the traffic packets, select whichone of the output ports the packet is to be output from onto the pathcoupled to the one of the output ports, wherein each of the shapersindicate a congestion level of the output port coupled to the shaperthat corresponds to a quantity of the packets sent to the output port bythe link selection logic during a time period, and further wherein ifthe packet does not have the TCP format, the link selection logicselects the one of the output ports based on the congestion level ofeach of the output ports. In some embodiments, if the packet does havethe TCP format, the link selection logic selects the one of the outputports independent of the congestion level of each of the output ports.In some embodiments, if the packet does have the TCP format, the linkselection logic selects the one of the output ports based on a hash ofthe packet and an equal or weighted cost multipath selection protocol.In some embodiments, if the packet does not have the TCP format, thelink selection logic selects the one of the output ports according to ametric except the link selection logic will remove all of the outputports whose congestion level is above a congestion threshold value froma pool of the output ports that are able to be selected according to themetric. In some embodiments, if the packet does not have the TCP formatand all of the output ports have a congestion level that is above thecongestion threshold value, the link selection logic selects the one ofthe output ports according to the metric while including all of theoutput ports in the pool despite the congestion level of all of theoutput ports. In some embodiments, the metric is one of the groupconsisting of round robin, random, and smallest congestion level first.In some embodiments, each of the shapers comprise a phantom queue and acredit generator that deposits a credit into the phantom queue at apredefined credit deposit rate, wherein as each packet is output by oneof the output ports, the shaper coupled to the one of the output portsremoves one or more credits from the phantom queue of the shaper suchthat a total value of the removed credits is equal to or greater than asize of the packet. In some embodiments, the link selection logicdetermines the congestion level of each of the output ports based on thenumber of credits within the phantom queue coupled to the output port.In some embodiments, the microchip has a plurality of packet queues eachcoupled with one of the output ports such that the queues receive andqueue each of the packets to be output by the output ports. In someembodiments, the link selection logic determines the congestion level ofeach of the output ports based on a number of the packets within thepacket queue associated with the output port. In some embodiments, themicrochip further comprises one of more additional shapers such thateach of the additional shapers is coupled to one of the output ports,wherein each of the additional shapers indicate an additional congestionlevel of the output port coupled to the additional shaper thatcorresponds to the quantity of the packets sent to the output port bythe link selection logic during the time period, and further wherein ifthe packet does not have the TCP format, the link selection logicselects the one of the output ports based on the congestion level andthe additional congestion levels of each of the output ports.

A third aspect is directed to a method of dynamic load balancing withina dynamic load balancing system. The method comprises receiving aplurality of traffic packets with link selection logic on a processingmicrochip having a plurality of shapers and a multipath interface groupincluding a plurality of paths for outputting packets from themicrochip, wherein each of the paths is coupled to an output port of themicrochip and each of the shapers is coupled to one of the output portsand monitors the outputting of the packets out of the output port todetermine whether a rate of data output by the output port is above adata output rate threshold, for each of the traffic packets, determiningwhether the packet has a transmission control protocol (TCP) format withthe link selection logic and for each of the traffic packets, selectingwhich one of the output ports the packet is to be output from onto thepath coupled to the one of the output ports with the link selectionlogic, wherein each of the shapers indicate a congestion level of theoutput port coupled to the shaper that corresponds to a quantity of thepackets sent to the output port by the link selection logic during atime period, and further wherein if the packet does not have the TCPformat, the link selection logic selects the one of the output portsbased on the congestion level of each of the output ports. In someembodiments, the method further comprises, if the packet does have theTCP format, selecting the one of the output ports independent of thecongestion level of each of the output ports with the link selectionlogic. In some embodiments, the method further comprises, if the packetdoes have the TCP format, selecting the one of the output ports based ona hash of the packet and an equal or weighted cost multipath selectionprotocol with the link selection logic. In some embodiments, the methodfurther comprises, if the packet does not have the TCP format, selectingthe one of the output ports according to a metric with the linkselection logic wherein the link selection logic removes all of theoutput ports whose congestion level is above a congestion thresholdvalue from a pool of the output ports that are able to be selectedaccording to the metric. In some embodiments, the method furthercomprises, if the packet does not have the TCP format and all of theoutput ports have a congestion level that is above the congestionthreshold value, selecting the one of the output ports according to themetric with the link selection logic while including all of the outputports in the pool despite the congestion level of all of the outputports. In some embodiments, the metric is one of the group consisting ofround robin, random, and smallest congestion level first. In someembodiments, each of the shapers comprise a phantom queue and a creditgenerator that deposits a credit into the phantom queue at a predefinedcredit deposit rate, further comprising as each packet is output by oneof the output ports, removing, with the shaper coupled to the one of theoutput ports, one or more credits from the phantom queue of the shapersuch that a total value of the removed credits is equal to or greaterthan a size of the packet. In some embodiments, the method furthercomprises determining the congestion level of each of the output portswith the link selection logic based on the number of credits within thephantom queue coupled to the output port. In some embodiments, theprocessing microchip further comprises a plurality of packet queues eachcoupled with one of the output ports such that the queues receive andqueue each of the packets to be output by the output ports. In someembodiments, the method further comprises determining the congestionlevel of each of the output ports with the link selection logic based ona number of the packets within the packet queue associated with theoutput port. In some embodiments, the processing microchip has one ormore additional shapers such that each of the additional shapers iscoupled to one of the output ports and monitors the outputting of thepackets out of the output port to determine whether the rate of dataoutput by the output port is below an additional data output ratethreshold, and further wherein each of the additional shapers indicatean additional congestion level of the output port coupled to theadditional shaper that corresponds to the quantity of the packets sentto the output port by the link selection logic during the time period,and further wherein if the packet does not have the TCP format, theselecting of the one of the output ports is based on the congestionlevel and the additional congestion levels of each of the output ports.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a dynamic load balancing system 100 according to someembodiments.

FIG. 2 illustrates a method of dynamic load balancing within a dynamicload balancing system according to some embodiments.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, numerous details are set forth forpurposes of explanation. However, one of ordinary skill in the art willrealize that the invention can be practiced without the use of thesespecific details. Thus, the present invention is not intended to belimited to the embodiments shown but is to be accorded the widest scopeconsistent with the principles and features described herein.

Embodiments are directed to a data processing system that comprises aphantom queue for each of a plurality of output ports each associatedwith an output link for outputting data. The phantom queuesreceive/monitor traffic on the respective ports and/or the associatedlinks such that the congestion or traffic volume on the outputports/links is able to be determined by a congestion mapper coupled withthe phantom queues. Based on the determined congestion level on each ofthe ports/links, the congestion mapper selects one or more non or lesscongested ports/links as destination of one or more packets. A linkselection logic element then processes the packets according to theselected path or multi-path thereby reducing congestion on the system.For example, when a current port/link is determined to be congested,packets are able to be re-routed to one or more of the other links/portsuntil the current port/link is no longer congested.

In some embodiments, the non-congested ports are selected by maskinglinks to congested ports. In some embodiments, the non-congested portsare determined based on their congestion level value being below acongestion threshold value and the congested ports are determined basedon their congestion level being above the congestion threshold value ora different threshold value. In some embodiments, a link/port isdetermined to be congested if a bucket of the associated phantom queueis empty and/or out of credits for outputting the traffic packets.Alternatively, or in addition, a link/port is determined to be congestedbased on the queue fill level for the port/link. In some embodiments,TCP traffic is not enabled for the dynamic load balancing of the systemsuch that the traffic is able to ignore congestion levels and thus isnot directed to different ports/links by the congestion mapperregardless of the congestion state. In some embodiments, non-TCP trafficis enabled for the load balancing of the system such that it is able tobe routed to different ports/links based on the congestion levels by thecongestion mapper. Alternatively, both the TCP and the non-TCP trafficis enabled for the load balancing of the system such that it is able tobe routed to different ports/links based on the congestion levels by thecongestion mapper. In some embodiments, if selection of one of aplurality of non-congested ports is required, the ports/links are ableto be selected randomly, in a round robin order, based on the level ofcongestion (e.g. which has the least current congestion), and/oraccording to other types of selection priority protocols. In someembodiments, one or more of the phantom queues are able to be replacedand/or supplemented with a traffic shaper. As a result, the systemprovides the advantage of considering phantom queue indications ofcongestion levels to dynamically balancing output port packet loads fornon-TCP traffic while disregarding phantom queue indications ofcongestion levels when distributing TCP traffic (e.g. staticallybalancing output port packet loads for TCP traffic).

FIG. 1 illustrates a dynamic load balancing system 100 according to someembodiments. As shown in FIG. 1, the dynamic load balancing system 100is able to be located within and/or stored on one or more processingmicrochips 102 (e.g. one or more software-defined network microchips,datacenter switch, ethernet switch). Alternatively, the system 100 isable to be located within and/or stored on one or more components of aprocessing circuit. The dynamic load balancing system 100 comprises aplurality of output ports 104, output paths 106, shapers 110, packetqueues 112 and link selection logic 114. Although as shown in FIG. 1,the system 100 comprises two output ports 104, output paths 106, shapers110 and packet queues 112, more output ports 104, output paths 106,shapers 110 and/or packet queues 112 are contemplated. Further, thesystem 100 is able to comprise more or less components. For example, insome embodiments the packet queues 112 are able to be omitted.Additionally, in some embodiments one or more of the output ports 104are able to each have a plurality of shapers 110 and/or packet queues112 operably coupled therewith.

The plurality of output ports 104 are each associated with one of theoutput path 106, which together form a multipath interface 108. Thus,packets that exit the chip 102 via one of the output ports 104 willtravel on the output path 106 associated with the output port 104. Insome embodiments, one or more of the output ports 104 are physical portsof the microchip 102. Alternatively, one or more of the output ports 104are able to be virtual ports of the microchip 102. Each one of theshapers 110 is operably coupled a different one of the packet queues 112and/or a different one of the output ports 104 such that each link orpath 106 is associated with a set of one queue 112, one shaper 110 andone port 104. Alternatively, as described above, a group of a pluralityof shapers 110 is able to be operable coupled to each of the packetqueues 112 and/or the output ports 104 such that each link or path 106is associated with a set of one queue 112, a group of shapers 110 andone port 104. As a result, for each of the output ports 104, the packetqueue 112 coupled to that port 104 is able to receive and buffer packetsthat are to be sent to the port 104 until the port 104 is ready tooutput them. For example, the queue 112 is able to receive packets asrouted by the link selection logic 114 and buffer the packets accordingto a first in first out (FIFO) or other buffering system until they areready to be received by the corresponding output port 104. Also for eachof the output ports 104, the shaper 110 coupled to that port 104 is ableto shape or control the packet rate (e.g. number of packets/time) of thepacket traffic traveling out of the output port 104. In particular, theshaper 110 is able to comprise a credit generator 110 a and a phantomqueue 110 b, wherein the credit generator 110 a fills the phantom queue110 b with credits at a predetermined credit rate and the shaper 110must remove one of the credits each time the shaper 110 permits a numberof packets having a size equal to or less than a value of the credit orcredits to be output through the corresponding output port 104. Forexample, if each credit is worth 256 bytes, the shaper 110 must removeone credit before permitting one or more packets whose size togetherequal the 256 bytes (i.e. the value of the credit). Correspondingly, ifeach credit is worth 256 bytes and the packet to be transmitted has asize of 300 bytes, the shaper 110 must wait for at least two credits toaccumulate within the queue 110 b before permitting the packet to beoutput and removing two of the at least two credits. Consequently, theshaper 110 is able to limit the maximum output rate of the packets outof the output port 104 because if there are no credits remaining in thephantom queue 110 b (because they all have previously been removed andthe next credit has yet to be deposited by the credit generator 110 a)the shaper 110 will prevent any further packets from being output untila new credit is available. On the other hand, if there are less packetsbeing selected for output via the port 104 (and therefore input by thepacket queue 112) than the value of the number of credits beingdeposited, the phantom queue 110 b is able to fill up with extra credits(that cannot be used because there are no packets to output) until thephantom queue 110 b is completely full. In this manner, the fill levelof each of the phantom queues 110 b is able to indicate a congestionlevel of the associated ports 104, wherein the fuller the phantom queue110 b the lower the congestion level of the port 104 and vice versa.

In some embodiments, the shapers 110 are able to be passive in that theydo not enforce restricting traffic or packet transmission to the shaperrate, rather they only passively monitor the rate of the packet trafficto detect when a congestion level is reached and then signal thatinformation to the selection logic. Further, in some embodiments whereinone or more groups of shapers 110 are each coupled to different singleoutput ports 104, each shaper 110 of the groups is able to have a creditgenerator 110 a that generates credits at a different rate and/or of adifferent size than the other credit generators 110 a of the othershapers 110 in the group. As a result, the different shapers 110 willeach have different phantom queue fill levels (i.e. indicate differentcongestion levels) based on the rates that credits are produced by theseparate credit generators 110 a in comparison with the rate thatpackets are being output via the associate output port 104. Thus, insuch embodiments, multi-level congestion indications (e.g. one for eachshaper 110 in the group) are able to be provided to the link selectionlogic 114 for each port 104 coupled with one of the groups of shapers110.

The link selection logic 114 is coupled with or is provided access toinput traffic packets 116 and each path 106 including the associatedport 104, shaper 110 and packet queue 112. As a result, the selectionlogic 114 is able to input or access traffic in the form of packets thatenter the system 100 and phantom queue vectors from the shapers 110indicating the current number of credits (e.g. a congestion level)within each of the phantom queues 110 b, and further able to determinewhich of the paths 106 and/or ports 104 each of the packets are outputfrom by the system 100. In particular, upon determining whether an inputpacket is a TCP or non-TCP format packet, the link selection logic 114is able to use a TCP selection metric to select one of the ports 104from which to output the input packet determined to be a TCP or TCPformat packet. For example, the TCP metric is able to be a weighted orequal cost multipath metric that selects a port 104 based on a hash orother representation of the TCP packets in order to attempt to maintainthe order of the sequence of the TCP packets. Alternatively, the TCPmetric is able to be other types of selection metrics that prioritizemaintaining the sequence of the TCP packets.

In contrast, if the input packet is determined to be a non-TCP or TCPformat packet, the link selection logic 114 is able to use a non-TCPselection metric and the phantom queue vectors to select one of theports 104 from which to output the input packet. Specifically, the linkselection logic 114 is able to input or review the latest phantom queuevector and remove any of the ports 104 whose vector value (or congestionlevel or phantom queue 110 b fill level) indicates a level of congestionthat exceeds a predetermined congestion threshold from the pool of ports104 that are able to be selected by the non-TCP selection metric. Then,based on this remaining pool of the ports 104, the link selection logic114 is able to select the one of the ports 104 from which to output theinput packet based on the non-TCP selection metric. Alternatively, theTCP selection metric is able to be used based on the remaining pool ofports 104. As a result, heavily congested ports 104 are prohibited fromselection by the selection logic 114 until their congestion level fallsback below the threshold thereby dynamically balancing the traffic loadon the ports 104 for the non-TCP traffic.

In some embodiments, the non-TCP selection metric is able to be the port104 whose vector value indicates the lowest level of congestion. Inparticular, in the case wherein a group of shapers 110 produce aplurality of congestion levels for each of the ports 104, the port 104with the lowest congestion level is able to be determined based on whichport 104 has the least number of shapers whose congestion level is abovethe threshold. In other words, in such embodiments the number of shapers110 of each of the groups of shapers 110 that indicate a congestionlevel above the threshold is able to be used by the selection logic 114to determine which port 104 to select and/or which ports 104 to removefrom the pool of selectable ports 104. Alternatively, the non-TCPselection metric is able to be a random, round robin or other scheduleof selecting one of the pool of ports 104.

In the case where based on the vector values the congestion levels ofall of the ports 104 of the multipath interface 108 exceed thecongestion threshold, the link selection logic 114 is able to add all ofthe ports 104 back into the pool (despite their congestion levels) andbased on this full pool of the ports 104 select the one of the ports 104from which to output the input packet based on the non-TCP selectionmetric. Alternatively, the TCP selection metric is able to be used insuch a case based on the full pool of ports 104. Thus, in any case thesystem provides the advantage of considering phantom queue 110 bindications of congestion levels to dynamically balancing output port104 packet loads for non-TCP traffic while disregarding phantom queue110 b indications of congestion levels when distributing TCP traffic(e.g.

statically balancing output port 104 packet loads for TCP traffic). Insome embodiments, each shaper 110 is subject to the same creditgeneration rate (e.g. congestion threshold). Alternatively, one or moreof the shapers 110 are able to be subject to different credit generationrates (e.g. congestion thresholds). In some embodiments, as shown inFIG. 1, the link selection logic 114 is able to comprise a firstcomponent 114 b that receives the phantom queue vectors and performs thelink selection for the non-TCP traffic and a second component 114 b thatperforms the link selection for the TCP traffic. Alternatively, thefirst and second components 114 a, 114 b are able to be combined as asingle component 114. In some embodiments, the determination whether thetraffic is TCP or non-TCP is able to be omitted and instead all trafficis able to be subject to the non-TCP selection metric as if it were allnon-TCP traffic as described above.

In some embodiments, other factors are able to be considered for non-TCPtraffic before removing ports 104 from the pool of ports 104 from whicha packet is output. For example, in addition to or in lieu of whetherthe congestion threshold is exceeded based on the phantom queue, thelink selection logic is able to determine and consider the current levelof fullness of packets of the associated packet queue 112. Inparticular, the packet queue fullness level is able to be compared tothe packet queue fullness threshold wherein a port 104 is removed fromthe pool only when both the packet queue fullness and the phantom queuethresholds have been exceeded, when at least one of the packet queuefullness and the phantom queue thresholds have been exceeded, or solelybased on when the packet queue fullness threshold has been exceeded.Alternatively or in addition, other factors such as quantized congestionnotification methods are able to be used to determine when to removeports 104 from the pool of ports.

FIG. 2 illustrates a method of dynamic load balancing within a dynamicload balancing system 100 according to some embodiments. As shown inFIG. 2, the link selection logic 114 accesses or receives a plurality oftraffic packets at the step 202. Then, for each of the traffic packets,the selection logic 114 determines whether the packet has a TCP formatat the step 204. Accordingly, for each of the traffic packets, the linkselection logic 114 selects which one of the output ports 104 the packetis to be output from onto the path 106 coupled to the one of the outputports 104, wherein if the packet does not have the TCP format, the linkselection logic 114 selects the one of the output ports 104 based on thecongestion level of each of the output ports 104 at the step 206.Specifically, the link selection logic 114 is able to determine thecongestion level of each of the output ports 104 based on the number ofcredits within the phantom queue 110 b coupled to the output port 104.Alternatively or in addition, the link selection logic 114 is able todetermine the congestion level of each of the output ports 104 based ona number of the packets within the packet queue 112 associated with theoutput port 104. As a result, the method is able to provide theadvantage of dynamically load balancing the outputting of the non-TCPtraffic based on port congestion level. If instead the packet does havethe TCP format, the link selection logic 114 is able to select the oneof the output ports 104 independent of the congestion level of each ofthe output ports 104. In other words, unlike non-TCP traffic, the system100 is able to recognize the preference for keeping TCP traffic insequence and thus does not apply the dynamic load balancing to its portselection for TCP traffic. As a result, the method further provides theadvantage of distinguishing between traffic types and applying differentport selection metrics based on the traffic type/format.

If the packet does have the TCP format, the selecting the one of theoutput ports 104 is able to be based on a hash of the packet and anequal or weighted cost multipath selection protocol. If the packet doesnot have the TCP format, the selecting the one of the output ports 104is able to be according to a non-TCP metric, wherein the link selectionlogic 114 removes all of the output ports 104 whose congestion level isabove a congestion threshold value from a pool of the output ports 104that are able to be selected according to the non-TCP metric. In someembodiments, the non-TCP metric is one of the group consisting of roundrobin, random, and smallest congestion level first. Alternatively, othermetrics are able to be used and/or a combination of round robin, random,and smallest congestion level first wherein the combined metrics areprioritized and implemented according to the priority wherein the nextmetric is used to break ties of the previous metric. Also, in someembodiments if the packet does not have the TCP format and all of theoutput ports 104 have a congestion level that is above the congestionthreshold value, the link selection logic 114 selects the one of theoutput ports 104 according to the metric while including all of theoutput ports 104 in the pool despite the congestion level of all of theoutput ports 104. Thus, the method provides the advantage of ensuringthe packet flow is not halted in the case that all the ports 104 areabove the congestion threshold.

Accordingly, the dynamic load balancing system provides the advantage ofdistinguishing between traffic types and applying different portselection metrics based on the traffic type/format. Further, the systemprovides the advantage of considering phantom queue indications ofcongestion levels to dynamically balancing output port packet loads fornon-TCP traffic while disregarding phantom queue indications ofcongestion levels when distributing TCP traffic (e.g. staticallybalancing output port packet loads for TCP traffic). Moreover, thesystem provides the advantage of ensuring the packet flow is not haltedin the case that all the ports are above the congestion threshold.Therefore, the dynamic load balancing system described herein hasnumerous advantages.

One of ordinary skill in the art will realize other uses and advantagesalso exist. While the invention has been described with reference tonumerous specific details, one of ordinary skill in the art willrecognize that the invention can be embodied in other specific formswithout departing from the spirit of the invention. For example,although the system described herein illustrates a single multipathinterface 108, a plurality of multipath interfaces 108 are contemplatedwherein each packet is assigned to one of the interfaces and is thensent to one of the ports 104 of that interface 108 as described above.As another example, although the different methods described hereindescribe a particular order of steps, other orders are contemplated aswell as the omission of one or more of the steps and/or the addition ofone or more new steps. Moreover, although the methods above aredescribed herein separately, one or more of the methods are able to becombined (in whole or part). Thus, one of ordinary skill in the art willunderstand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims. Additionally, it should be noted that, unlike policers, shapers110 do not drop any packets in order to control the output rate of aport 104. Instead, shapers 110 only delay the packets to ensure themaximum output rate is not exceeded.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. Thus, one of ordinary skill in the artwill understand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims.

We claim:
 1. A dynamic load balancing system on a microchip, the systemcomprising: a plurality of output ports; and link selection logic thatreceives packets and, for each of the packets, selects which one of theoutput ports the packet is to be output from, wherein for each of thepackets: if the packet has a transmission control protocol (TCP) format,the link selection logic selects the one of the output ports independentof a congestion level of each of the output ports; and if the packetdoes not have the TCP format, the link selection logic selects the oneof the output ports based on the congestion level of each of the outputports.
 2. The system of claim 1, wherein if the packet does have the TCPformat, the link selection logic selects the one of the output portsbased on a hash of an input traffic packet and an equal or weighted costmultipath selection protocol.
 3. The system of claim 2, wherein if thepacket does not have the TCP format, the link selection logic selectsthe one of the output ports according to a metric except the linkselection logic will remove all of the output ports whose congestionlevel is above a congestion threshold value from a pool of the outputports that are able to be selected according to the metric.
 4. Thesystem of claim 3, wherein if the packet does not have the TCP formatand all of the output ports have the congestion level that is above thecongestion threshold value, the link selection logic selects the one ofthe output ports according to the metric while including all of theoutput ports in the pool despite the congestion level of all of theoutput ports.
 5. The system of claim 4, wherein the metric is one of agroup consisting of round robin, random, and smallest congestion levelfirst.
 6. The system of claim 5, further comprising a plurality ofshapers, wherein each of the shapers comprise: a phantom queue; and acredit generator that deposits a credit into the phantom queue at apredefined credit deposit rate; wherein as each of the packets is outputby one of the output ports, the shaper coupled to the one of the outputports removes one or more credits from the phantom queue of the shapersuch that a total value of the removed one or more credits is equal toor greater than a size of the packet.
 7. The system of claim 6, whereinthe link selection logic determines the congestion level of each of theoutput ports based on the number of credits within the phantom queuecoupled to the output port.
 8. The system of claim 7, further comprisinga plurality of packet queues each coupled with one of the output portssuch that the queues receive and queue each of the packets to be outputby the output ports.
 9. The system of claim 8, wherein the linkselection logic determines the congestion level of each of the outputports based on a number of the packets within a packet queue associatedwith the output port.
 10. The system of claim 9, further comprising oneor more additional shapers, wherein each of the additional shapers iscoupled to one of the output ports and monitors outputting of thepackets out of the output port to determine whether a rate of dataoutput by the output port is above an additional data output ratethreshold, and further wherein each of the additional shapers indicatean additional congestion level of the output port coupled to theadditional shaper.
 11. A link selection logic element stored on anon-transitory computer-readable medium of a microchip having aplurality of output ports, wherein and configured to: receive aplurality of packets input by the microchip; and for each of thepackets, select which one of the output ports the packet is to be outputfrom, wherein if the packet has transmission control protocol (TCP)format, selection of the one of the output ports is independent of acongestion level of each of the output ports, and if the packet does nothave the TCP format, selection of the one of the output ports is basedon the congestion level of each of the output ports.
 12. The linkselection logic element of claim 11, wherein if the packet does have theTCP format, the link selection logic element selects the one of theoutput ports based on a hash of the packet and an equal or weighted costmultipath selection protocol.
 13. The link selection logic element ofclaim 12, wherein if the packet does not have the TCP format, the linkselection logic element selects the one of the output ports according toa metric except the link selection logic element will remove all of theoutput ports whose congestion level is above a congestion thresholdvalue from a pool of the output ports that are able to be selectedaccording to the metric.
 14. The link selection logic element of claim13, wherein if the packet does not have the TCP format and all of theoutput ports have the congestion level that is above the congestionthreshold value, the link selection logic element selects the one of theoutput ports according to the metric while including all of the outputports in the pool despite the congestion level of all of the outputports.
 15. The link selection logic element of claim 14, wherein themetric is one of a group consisting of round robin, random, and smallestcongestion level first.
 16. The link selection logic element of claim15, wherein the microchip comprises a plurality of shapers and each ofthe shapers comprise: a phantom queue; and a credit generator thatdeposits a credit into the phantom queue at a predefined credit depositrate; wherein as each of the packets is output by one of the outputports, the shaper coupled to the one of the output ports removes one ormore credits from the phantom queue of the shaper such that a totalvalue of the removed one or more credits is equal to or greater than asize of an input traffic packet.
 17. The link selection logic element ofclaim 16, wherein the link selection logic element determines thecongestion level of each of the output ports based on a number ofcredits within the phantom queue coupled to the output port.
 18. Thelink selection logic element of claim 17, wherein the microchip has aplurality of packet queues each coupled with one of the output portssuch that the queues receive and queue each of the packets to be outputby the output ports.
 19. The link selection logic element of claim 18,wherein the link selection logic element determines the congestion levelof each of the output ports based on a number of the packets within apacket queue associated with the output port.
 20. The link selectionlogic element of claim 19, wherein the microchip further comprises oneor more additional shapers such that each of the additional shapers iscoupled to one of the output ports, wherein each of the additionalshapers indicate an additional congestion level of the output portcoupled to the additional shaper, and further wherein if the packet doesnot have the TCP format, the link selection logic element selects theone of the output ports based on the congestion level and the additionalcongestion levels of each of the output ports.
 21. A method of dynamicload balancing within a dynamic load balancing system, the methodcomprising: receiving a plurality of packets with link selection logicon a microchip having a plurality of output ports, wherein; and for eachof the packets, selecting which one of the output ports the packet is tobe output from with the link selection logic, wherein if the packet hasa transmission control protocol (TCP) format, selection of the one ofthe output ports is independent of a congestion level of each of theoutput ports, and if the packet does not have the TCP format, selectionof the one of the output ports is based on the congestion level of eachof the output ports; wherein the congestion level of the output port isindicated by a shaper coupled to the output port.
 22. The method ofclaim 21, further comprising, if the packet does have the TCP format,selecting the one of the output ports based on a hash of the packet andan equal or weighted cost multipath selection protocol with the linkselection logic.
 23. The method of claim 22, further comprising, if thepacket does not have the TCP format, selecting the one of the outputports according to a metric with the link selection logic wherein thelink selection logic removes all of the output ports whose congestionlevel is above a congestion threshold value from a pool of the outputports that are able to be selected according to the metric.
 24. Themethod of claim 23, further comprising, if the packet does not have theTCP format and all of the output ports have the congestion level that isabove the congestion threshold value, selecting the one of the outputports according to the metric with the link selection logic whileincluding all of the output ports in the pool despite the congestionlevel of all of the output ports.
 25. The method of claim 24, whereinthe metric is one of a group consisting of round robin, random, andsmallest congestion level first.
 26. The method of claim 21, wherein themicrochip includes a plurality of shapers and each of the shaperscomprise: a phantom queue; and a credit generator that deposits a creditinto the phantom queue at a predefined credit deposit rate; furthercomprising as each of the packets is output by one of the output ports,removing, with the shaper coupled to the one of the output ports, one ormore credits from the phantom queue of the shaper such that a totalvalue of the removed one or more credits is equal to or greater than asize of the packet.
 27. The method of claim 26, further comprisingdetermining the congestion level of each of the output ports with thelink selection logic based on a number of credits within the phantomqueue coupled to the output port.
 28. The method of claim 27, whereinthe microchip further comprises a plurality of packet queues eachcoupled with one of the output ports such that the queues receive andqueue each of the packets to be output by the output ports.
 29. Themethod of claim 28, further comprising determining the congestion levelof each of the output ports with the link selection logic based on anumber of the packets within a packet queue associated with the outputport.
 30. The method of claim 29, wherein the microchip has one or moreadditional shapers such that each of the additional shapers is coupledto one of the output ports and monitors outputting of the packets out ofthe output port to determine whether a rate of data output by the outputport is below an additional data output rate threshold, and furtherwherein each of the additional shapers indicate an additional congestionlevel of the output port coupled to the additional shaper, and furtherwherein if the packet does not have the TCP format, the selecting of theone of the output ports is based on the congestion level and theadditional congestion levels of each of the output ports.